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ABSTRACT 



This research investigated how different experts in a single domain chose their 
individual subjective evaluation criteria of a highly aggregate task based upon their 
individual differences. The Conning Officer Virtual Environment (COVE) was utilized 
to provide a domain of experts and a subjectively evaluated task. 116 expert ship- 
handlers were investigated to understand how their personality affects their evaluation of 
a novice performing an underway replenishment (UNREP). The experts were issued a 
survey that inventoried their personality, UNREP evaluation criteria, and ship handling 
style. In general, the participant experts were lower in Neuroticism and higher in 
Extraversion and Conscientiousness than the average adult. Extraversion appeared to be 
correlated with the expert’s desire to use Sensory Input as a critical evaluation criterion 
(p = .18) while Openness was correlated with Analytical Input (p = .16) and UNREP style 
(p = .16) as critical evaluation factors. Also correlated with UNREP style was 
Agreeableness (p = .16). Finally, the expert’s level of Conscientiousness correlated with 
the critical evaluation criteria of Analytical Input (p = .17) and Sensory Input (p = .39). 
Results from this research provide insight to the link between observed behavior and its 
subjective evaluation and will allow COVE’s programmers to develop an Intelligent 
Tutoring System (ITS) that will customize the automated training process. 
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I. INTRODUCTION 



A. MOTIVATION 

The military serv^ices have historically been an apprenticeship system. Beginning 
with the early days of sail, United States Naval Midshipmen would spend several years 
serving aboard a single ship with a single captain. Under the tutelage and guidance of his 
master captain, the apprentice midshipman would learn the art of sail and war. A 
midshipman would be promoted to the rank of an officer of the line only after gaining his 
captain’s full trust and confidence in his knowledge and abilities. This type of highly 
specialized training required vast resources and the dedication of numerous personnel. 

Whether a sailor, marine, soldier, or airman, the United States Military warrior of 
the new millennium is required to do more with less. Fewer troops, weapons, training 
time, and fiscal resources are requiring the armed forces to re-evaluate every facet of 
operations. In particular, budget draw downs and the increasing complexity of hardware 
necessitates the need to create cost-effective training alternatives. As computing power 
and speed increase, the desire for utilizing computers as a beneficial training tool also 
increases. Using modem computers’ ever increasing high fidelity virtual environments 
(VEs) as a training tool for performing spatial and cognitive tasks are a particular area of 
increasing demand since VEs provide the potential capability for a trainee to practice and 
master complex and highly dangerous tasks safely, efficiently, and economically 
[CAIR96]. 
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B. 



OBJECTIVE 



“The best known generalization in human learning is that practice makes perfect” 
[ANNE89]. The caveat to this cliche is that the student is practicing the right task in the 
right ways. The ability to properly react to any situation requires expert guidance and 
proper intervention at critical points during training. Without a good teacher, practice 
alone is not always enough to become competent at a complex task. Even more 
devastating is the possibility that the student will get worse at the learned task and 
experience a negative training transfer [BOLD87]. 

While VE is a relatively new training tool, a VE training system is not 
pragmatically different from any previous generation of training tool. VE training must, 
just like any other training system, provide students with the skills and knowledge 
required to meet the demands of the trained task and the needs of the overriding 
organization [CAIR96]. While any form of training requires several key components to 
be effective, one of the most essential steps to developing a successful training program is 
providing quality feedback via instruction and evaluation. 

The Corming Officer Virtual Environment (COVE) Ship Handling Trainer is one 
example of a VE that provides an economically attractive alternative to traditional 
methods of practicing ship handling while providing an integrated means of instruction. 
COVE, currently under development by Naval Air Warfare Training Systems Division 
(NAWC-TSD), simulates ship-handling scenarios where the trainee is immersed in a VE, 
complete with an integrated intelligent tutoring system (ITS), in the form of a simulated 
interactive commanding officer. COVE is a flexible and portable unit that is intended to 
build and reinforce ship-handling skills with minimal requirements for instructor 



2 



intensiveness and costly ship resources [MEAD99]. If the deployed implementation of 
COVE is successful, Junior Officers (JOs) will have an opportunity to develop basic 
skills and practice difficult scenarios in a controlled environment without the need to 
have entire ships at sea, saving time, dollars, and possibly even lives. 

“Providing the trainee with knowledge of the results is one of the most common 
training program interventions and one which is generally believed to have a powerful 
effect on learning” [ANNE89], In the case of COVE, quality feedback to the trainee 
requires the ITS to be more than a scripted set of rules. The ITS must provide immediate 
guidance and feedback that is accurate and meets accepted standards, just as a 
Commanding Officer (CO) would at sea, else risk the loss of valuable training time and a 
possible negative transfer training experience [TENN99]. 

ITS feedback should both conform to accepted, safe practices and the 
requirements of the JO’s CO. However, the dynamics of handling a ship at sea combined 
with individual differences of COs makes it difficult to have a single standardized set of 
feedback responses. Just as the original shipmasters trained their apprentices uniquely, 
today’s COs train their JOs according to their predilections. Different COs will have 
different benchmarks based upon their own style of expertise, experiences, and 
personality, resulting in different COs evaluating the same evolution differently [NPS99]. 
In order to gain maximum benefit for the fleet, COVE’s ITS must be flexible enough to 
meet the needs of the different fleet experts. 

While a prime example, COVE is just a single example of a trainer that requires 
extensive knowledge and that has infinitely many ways to arrive at a "correct" solution 
that is "correct" only in the eyes of the evaluator. Topics easily range from driving ships. 



3 



to land navigation, to philosophy. Essentially, anything that involves asynchronous 
student-paced instruction and training of a highly complex aggregate task that involves 
subjectivity in evaluation can benefit from the relationships explored by the COVE ITS 
and student. 

This study investigates how simulator performance evaluation should be modeled 
based upon the personality composition of the evaluating expert. With respect to the 
COVE trainer, it is desired to understand the different evaluation criteria used by different 
COs and its relationship to their personality, ultimately resulting in a more accurate ITS 
where the Virtual CO (VCO) could approximate a wide range of real world COs. 



To be true to form, one would have to have the many different styles of 
COs within the system and the ability to choose which one you need. At 
the one end would be the screamer that we may be most familiar with who 
will throw you off the bridge if you go too far, and at the other end would 
be the true mentor who lets you get to the point of no return only to help 
you avoid the collision that you thought was inevitable. 

Commanding Officer of an LPD 



This accurate modeling results in more effective trainer time by teaching the JO the same 
lessons his real world CO would teach, increasing the effectiveness and overall benefit of 
the trainer. 
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C. THESIS QUESTIONS 



The following questions are addressed in this thesis: 



• Is there a relationship between one’s personality and one’s 
expertise? 

• If such a relationship exists, can it be quantified? 

• Does it extend beyond individual expertise to the expert’s 
evaluation of others’ performance? 

• What is the range of characteristics of different ship driving styles? 

Addressing these questions is the first step in building a more accurate ITS for 
COVE. Since this research is only the initial exploration between human behavior and 
expert evaluation, it is intended to begin the initial compilation of a database for the 
COVE ITS. Understanding the answers to the aforementioned questions will provide 
cove’s its programmers with a realistic model to base various prototypical VCOs 
upon. Furthermore, these answers also lay the foundation for automating the relationship 
and increasing the fidelity between instructor and student in any VE with an ITS. This 
added insight will help mate the ITS with the student, potentially increasing positive 
training transfer for any VE training system. 



D. APPROACH 

In order to answer the questions posed by this research, knowledge about the 
relationship between experts and novices is required. Along with the nature of expertise. 
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knowledge about personality and its measurement must be obtained. Furthermore, the 
scope of this work requires an understanding of the sociological domain within which the 
experts and novices reside. 

Naval officers achieve the prestige of command only by displaying sustained 
superior performance, primarily at sea. As the senior ship driver aboard, and the one 
person ultimately responsible for any mishap, the Commanding Officer (CO) is the 
resident ship-handling expert. How a ship is driven by any of the ship’s officers is a 
direct statement about the ship handling abilities of its CO. 

Few evolutions make or break a CO’s reputation like the UNREP approach to the 
auxiliary replenishment ship since the approach is a calling card for the CO’s style and 
ability. While all UNREP experts achieve the same ultimate end goal of coming along 
side the replenishment ship, different COs accomplish this task differently. Some prefer 
to “John Wayne” with large speed differentials and small distances from “rubbing paint” 
while others prefer more of a slow and steady approach. Some COs base decisions on 
aggregate big picture data while others require more specific input. 

While the UNREP is one of the greatest showcases of skills for the surface 
warfare officer, it is also one of the most dangerous where the potential for loss of life 
and damage to not only one but also two ships is extremely high. The ability to actually 
practice this formidable task at sea is limited and can quickly evolve into a situation too 
complex for a junior officer to handle. These criteria result in good VE training being 
crucial and indicate that UNREP is an excellent VE candidate since it allows the 
opportunity for officers to develop prerequisite skills in a safe and controlled environment 
with minimal operating cost. 
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Because of all of the aforementioned factors, UNREP was the vehicle for this 



research and analysis. Its importance in the sociological domain of the ship driver also 
makes it suitable to determine the correlations with ship driver personality. An analysis , 
of the expert evaluation of an UNREP approach as performed by a less-experienced JO 
was correlated with the personality of the expert to answer the thesis questions. Results 
can be directly applied to COVE’s existing ITS during COVE simulated UNREPs. 



E. SUMMARY OF CHAPTERS 



The remainder of this thesis is broken down into the following chapters: 



• Chapter II provides background information on the mental and 
behavioral processes invoked during UNREP and other complex 
tasks. First, a review of the basic components of an UNRJEP is 
performed followed by a summary of COVE and ITS previous 
research. Next, a more in-depth view of the differences between 
experts and novices is explored in order to understand the 
differences between COs and JOs. Additionally, individual 
differences and their effects on decision making are explored. 
Finally, personality and its measurement are discussed in order to 
understand how individual expert COs are different from each 
other. 

• Chapter III discusses the apparatus utilized to gather information 
for this research. Reasons for selection, design, and development 
are covered for the two data collection tools, the NEO-FFI 
personality inventory and the Ship Handler Evaluation Survey. 

• Chapter IV delineates the methods utilized for data collection and 
analysis. An explanation detailing the administration of the survey 
is provided in addition to a summary of the construction of the data 
package. 
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• Chapter V summarizes the results from the data collection and 
analysis. Results are provided detailing the personality 
characteristics of the participant expert ship handlers, the critical 
evaluation criteria utilized by expert ship-handlers for evaluation of 
novice JOs, and significant correlations observed between 
personality and critical evaluation criteria. 

• Chapter VI presents a final discussion of the results of this thesis 
and describes areas requiring further research. Answers to thesis 
questions proposed by this research are addressed in addition to the 
possible ramifications of this research. 
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II. BACKGROUND 



A. THE UNREP EVOLUTION 

The UNREP evolution, while complex and dangerous during execution, is a 
particularly straightforward task. Two ships, an approach vessel and a replenishment 
vessel execute the evolution. The approach vessel is a warship that requires 
replenishment of its fuel and or stores. The replenishment vessel is usually a refueling 
tanker. The overall goal of the evolution is for the approaching vessel to come within 
close proximity of the replenishing vessel and bring on fuel and other supplies with out 
any damage to personnel or equipment. 

The UNREP is composed of distinct phases consisting of preparations, waiting, 
approach, alongside, and breakaway. Figure 1 is a diagram depicting the different phases 
involved in a plausible UNREP scenario and highlights some of the distances involved 
between the two ships participating in an UNREP. The evolution actually starts hours 
before the actual transfer of supplies is executed by performing the preparation phase. 
Checks of ships systems and a pre-execution brief are performed on both ships to ensure 
that both the ships and crews are prepared to perform the actual task. 

The next phase, the waiting phase, is just prior to the commencement of the 
approach. During this phase, the approach vessel maneuvers to a waiting station where 
the approach vessel will perform its last checks and wait for a signal from the 
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replenishing vessel to commence the approach. The waiting station is usually an area 



approximately 1000 yards astern of the replenishment vessel. 




Figure 1 ; A Diagram of a possible UNREP evolution. 
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Once both ships are on an agreed upon course and speed and are ready, signals are 
made and the approach phase commences. During the approach phase, the approaching 
vessel maneuvers from waiting station to a position directly alongside the replenishment 
vessel. It is during the approach phase that the first interaction of physical forces occurs 
between the two vessels. 

Once the approach is made, lines connecting the two ships are secured and the 
approach phase transitions into the alongside phase. During the alongside phase, the 
approach vessel maintains a constant position relative to the replenishment vessel during 
the transfer of fuel and stores. Radio communication is maintained between both vessels 
during the alongside phase until transfer is complete between the replenishment vessel 
and the approach vessel. Transfer time primarily depends upon the amount of supplies to 
be transferred, but typically is less than an hour. 

Once all supplies are transferred, all connecting lines between the two ships are 
cast off, marking the beginning of the breakaway phase. During this final phase, the 
approach vessel maneuvers away from the replenishment vessel. Once clear of the 
replenishment vessel, the approach vessel is no longer restricted in its ability to maneuver 
and can proceed on its own independent course and speed. 



B. THE CONNING OFFICER VIRTUAL ENVIRONMENT AND 
INTELLIGENT TUTORING SYSTEM 

The COVE trainer is a direct evolvement of a previous project by NAWC-TSD, 
the Virtual Environment Submarine (VESUB) Simulator. VESUB was intended to 



11 



provide a means for submarine officers to practice surfaced submarine evolutions, in 
particular transit in and out of port, with out the need of a surfaced submarine. COVE 
combines some of the original VESUB visual simulation architecture with voice 
recognition and an integrated intelligent tutoring system. Ideally, COVE is a portable, 
low cost training solution that provides the user with a high fidelity synthetic ship driving 
experience and requires no operator monitoring or intervention [MEAD99]. 

Most previously implemented expert systems possess limited capability for 
diagnosis and feedback making them relatively unsuitable for training purposes 
[TENN99]. In order for an artificially intelligent (AI) training system to be successful, it 
must possess the capability to learn from experience by making human-like associations 
requiring a sense of appropriate output and understanding of needs, desires, and emotions 
[DREY96]. A possible architecture that meets these criteria incorporates adaptive 
technology into a pedagogical agent. An example of a first generation ITS is STEVE 
(Soar Training for Virtual Environments), which is currently under development by the 
Air Force Research Laboratory [TENN99]. STEVE is designed to be a modular agent 
implementation for the purpose of instruction in a variety of computer-based learning 
environments [JOHN98], 

The COVE trainer incorporates an ITS in the form of a Virtual Commanding 
Officer (VCO). The VCO is a pedagogical agent that instructs the JO on how to properly 
drive the ship during a ship handling evolution such as UNREP [TENN99]. Previous 
research investigated three possible profiles for the VCO consisting of a passive VCO, a 
proactive VCO, and an aggressive VCO. The classification of passive, proactive, or 
aggressive was based primarily on a CO’s predilection to recommend course and speed 
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changes while a JO was conning the ship during an UNREP evolution. The three 
classifications were chosen for their broad coverage of different ship driving styles and 
were intended to represent the majority of ship drivers in the fleet. 



C. DIFFERENCES BETWEEN EXPERTS AND NOVICES 

“. . . I want the JO to learn to drive the ship the way I drive the ship.” 

Member of SURFLANT staff addressing 
NAWC-TSD about COVE and ITS. 

While a CO will usually find other styles acceptable, he will prefer his JOs to 
drive the ship in a manner similar to his. Much like a father teaching his teenager how to 
drive a car, the expert CO will first instruct, and later expect the novice JO to analyze 
data and make decisions in the same fashion as the CO. These expectations are the basis 
for the expert CO’s evaluation of the novice JO and are shaped by the CO’s expertise. 

UNREPs are dynamic, complex tasks and UNREP skill cannot be neatly 
categorized under a single type of expertise. Expertise itself is diverse and is segregated 
into the four following categories: [CHI88] 



• PRACTICAL EXPERTISE that primarily deals with motor skills 
or mental skills. Examples of practical skills are typing, 
memorizing restaurant orders, or mental calculation. This type of 
expertise often allows for parallel thought processing. 

• PROBLEM SOLVING EXPERTISE requiring specific domain 
related knowledge. Examples of problem solving expertise are 
computer programming or solving physics problems. 
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• ILL-DEFINED EXPERTISE that requires decisions under 
uncertainty, such as when some uncontrolled intervening event 
occurs between the choice and the outcome. An example of an ill- 
defined problem is predicting stock market performance [CHI88]. 

• DIAGNOSTIC EXPERTISE where metacognition is required to 
accurately access the reason for a given circumstance or set of 
facts. An example of diagnostic expertise is properly accessing an 
illness or medical condition from x-rays or symptoms. 



All four of the previous expertise categories apply to an UNREP evolution. 
Commands are issued and executed with practical expertise. Estimating times and speeds 
in open-ocean utilize problem-solving techniques. Given the dynamic nature of an 
UNREP due to the uncontrollable forces of nature and the interactions of two separate 
independent ship drivers simultaneously, UNREPs require both ill-defined and diagnostic 
expertise. A novice JO must demonstrate proficiency of all types of expertise in order to 
receive a favorable evaluation of the UNREP evolution from his CO. 

Expert ship handlers usually distinguish themselves from novices by determining 
the quickest, most efficient courses of action, and when a ship handling evolution is 
getting out of control. In distinguishing themselves from novices, experts, regardless of 
the area of expertise, share common traits. These commonalties are summarized as: 



• Experts excel mainly at their own domains. 

• Experts perceive large meaningful patterns in their domains. 

• Experts quickly solve problems with little error. 

• Experts have superior short-term and long-term memory. 

• Experts see and represent a problem at a deep (more principled) 
level. 
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• Experts spend a great deal of time analyzing a problem 
qualitatively. 

• Experts have strong self-monitoring skills. 

These traits usually result in an expert performing a task quicker and with fewer errors 
[CHI88]. 

Even though experts distinguish themselves apart from novices in common ways, 
there are still wide variabilities amongst the experts themselves. These variabilities are 
unique to each expert and are often referred to as individual differences. These 
differences influence how the expert responds to situations, teaches his novices, and 
evaluates his trainees’ proficiency. Understanding individual differences of the CO are 
critical inputs to producing useful feedback for the JO. 



D. DECISION-MAKING AND INDIVIDUAL DIFFERENCES 

A study performed by the U.S. Army Research Institute for the Behavioral and 
Social Sciences assessing how senior Army officers made critical battlefield decisions 
discovered that not all experts analyze situations and make decisions the same way 
[COHE96]. Most experts generally fall into two completely different paradigms. Some 
experts follow an analytical approach where decision-making is characterized by 
attempting to use rational and computational methods. In contrast, a recognition expert 
would attempt to make decisions based on fitting the situation into a known pattern and 
responding with a familiar label or plan of action. 
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Another U.S. Army Research Institute for the Behavioral and Social Sciences 
study investigated the effects of expertise, cognitive style, and mission on what 
information is used by senior Army officers during tactical decision making in an attempt 
to develop a tactical decision aid [MICH88]. Their research indicated that a tactical 
decision aid must be adaptable to individual differences such as personality, cognitive 
style, and preferences for sensory modality and communication mode. These findings 
correlate with the research on how Army officers performed under stressful situations. 
Their findings showed that personality exhibited some consistent patterns of response to 
stressful situations. Their research assumed that there is a reciprocal causality between 
individual, situational, and response variables and that the way an individual responds to 
a situation is directly affected by the individual’s personality. 



E. THE FIVE FACTOR MODEL OF PERSONALITY AND FEEDBACK 

The UNREP is an extremely stressful evolution for both novice and expert. Since 
an expert CO will perform an UNREP based upon his expertise paradigm, which is 
shaped by his individual characteristics, it is necessary to study the COs personality. 
Personality is often explained by the Five-Factor Model (FFM), which describes 
personality in terms of five distinct personality traits. The “ FFM originated in initial 
works by Fiske (1949), Norman (1963), and Tuppes and Christal (1963), who produced a 
highly stable structure with five factors” [SALG97]. The FFM is extremely attractive due 
to its empirical roots. While most models are derived from theoretical perspectives, the 
lexical FFM has a theoretically neutral position [WIDI97]. 
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The Revised NEO Personality Inventory (NEO PI-R) is modeled after the FFM. It 
is a widely accepted measure of personality developed by Dr. Paul Costa and Dr. Robert 
McCrae, assesses personality in terms of Neuroticism, Extraversion, Openness, 
Agreeableness, and Conscientiousness. The five personality factors are described in the 
following way: 



• EXTRAVERSION is the factor that describes people who are rated 
by their peers as “sociable, fun-loving, affectionate, friendly, and 
talkative” [MCCR87] versus “reserved, timid, and quiet” 
[SALG97]. 

• People high in AGREEABLENESS are forgiving, lenient, 
sympathetic, agreeable, and softhearted, according to peer ratings 
[MCCR87]. Peers describe those low in Agreeableness in more 
negative terms: ruthless, uncooperative, suspicious, and stingy. 

• Peers describe people high in CONSCIENTIOUSNESS as careful, 
well organized, punctual, ambitious, and persevering [MCCR87]. 
Conscientiousness “includes both proactive (hardworking, 
ambitious) and inhibitive (dutiful, scrupulous) aspects” 
[MCCR89]. 

• People who score high on NEUROTIC ISM typically report 
negative emotions such as worry, insecurity, self-consciousness, 
and tempermentalness [MCCR87] whereas people with low 
Neuroticism are calm, self-confident, and cool [SALG97]. 

• The final factor in this model is OPENNESS . Adjectives from 
lexical studies that describe this factor include “original, 
imaginative, broad interests, and daring” [MCCR87]. “Openness 
defines individuals who are creative curious, and cultured versus 
practical with narrow interests. [SALG97]” 



The five factors of personality have implications for occupational performance 
and therapy. Most research studying the relationship between personality and job 
performance only attempts to correlate quality of job performance with personality 
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[CLON96]. Two mela-analytic studies by Barrick and Mount (1991) and Tett and 
Jackson (1991) find that Conscientiousness is the only predictor of quality of job 
performance [RUST99]. 

There is less research on the link between individual differences and method of 
task completion. A previous study showed correlation between individual differences 
and variability in expertise [NPS99]. Specifically, ship handler’s methods for performing 
an UNREP varied into two distinct categories, analogical or analytical. Whether a CO 
performed an UNREP in an analytical fashion, or an analogical fashion could be 
correlated with the CO’s personality and cognitive style. A study investigating the theory 
that personality is more differentiated at higher levels of ability discovered that some 
personality traits are statistically more variable for individuals at high versus low levels of 
ability [AUST97]. This research also showed relationships between types of judgment 
and FFM factors and Intelligence Quotient (IQ). 

Rust, 1999, investigated the ability of the FFM to predict supervisor’s ratings of 
performance. In his research. Rust administered the Orpheus; broad-spectrum work 
based personality questionnaire to employees. His findings showed a correlation between 
the FFM results of the self-evaluation Orpheus and appropriate supervisor ratings. In 
evaluating the FFM within the context of work based behavior: 

• High extroversion people are generally happier working with 
others while low extroverts tend to prefer work requiring 
independence. 

• High agreeableness results in individuals with a desire for a more 
cooperative, problem-solving approach the lower Agreeableness 
results in an ability to make tough decisions. 
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• People displaying high openness to experience seek alternative 
solutions and desire different methods, while low openness to 
experience individuals desire traditional approaches and respect 
established values. 

• Low neurotics tend to disregard feelings of others, perform better 
under stress, but tend to lack caution. 

• High conscientious people tend to excel at detailed tasks, but may 
become over involved in minutiae while low conscientiousness 
people have little patience for mundane tasks, and prefer to see the 
big picture. 

[RUST99] 

This research is important because personality is a factor for how a CO learns, and 
subsequently trains. The expert is more inclined to use teaching techniques in a manner 
that he understands the best. Previous research has attempted to correlate personality 
traits with various learning styles. This research indicated that Extroversion and 
Agreeableness are linked with more active types of learning [FURN96]. Therefore, it is a 
conclusion that Extroversion and Agreeableness could explain active forms of teaching. 
In the case of an UNREP, these personality traits could explain why some COs are more 
actively involved with the JO during the evolution than others are. 

Salgado analyzed three prior meta-analysis studying the relationship between 
personality and job criteria. In general, Salgado discovered that “Extraversion is a valid 
predictor of training proficiency (p = .26), as are Neuroticism (p = .07), Agreeableness 
(p = .10), and Openness to Experience (p = .25)” [SALG97]. Furthermore, personality 
compatibility between teacher and student will potentially affect the teacher’s evaluation 
of an evolution. Research demonstrates that students achieved higher levels in classes 
when teacher-student personality compatibility is high [FURN96]. This teacher and 
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student dynamic should also apply to VE training, and if properly modeled, will further 



increase the accuracy of the training feedback. 



F. PERSONALITY MEASUREMENT 

The assessment of personality is a major application of psychology to real world 
concerns and is extremely varied in its administration and utilization. Clinical 
psychologists evaluate a patient’s personality in an attempt to determine if the patient 
possesses abnormal symptoms or feelings. A school psychologist will assess a child’s 
personality in order to determine any causes of possible learning or adjustment problems. 
Counseling psychologists attempt to determine the best job for a particular person by 
matching the individual’s needs and interests with the requirements of the position. 
Finally, research psychologists assess the personalities of experiment participants to 
account for experimental behavior or correlate personality characteristics with other 
measures [SCHU90]. 

Regardless of the end goal, some assessment techniques are more objective while 
some techniques are wholly subjective and prone to bias. The best techniques possess 
standardization, reliability, and validity. Standardization insures consistency and 
uniformity of the procedures utilized for the test administration. Reliability insures 
consistency of results to the assessment device. Finally, validity insures that the test 
device results are an accurate measurement of the intended measured variables 
[SCHU90]. 
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Multiple methods exist to assess personality. A primary method is referred to as 
the self-report inventory method. In the self-report method, people report on themselves 
by answering questions about their feelings and behavior in a variety of simulated 
situations. The person taking the test must indicate how closely each item describes their 
own characteristics or how much they agree with each item. In general, self-report 
personality assessment methods are high in reliability and validity due to the standardized 
nature of administration, scoring, and evaluation of the results [SCHU90]. 

The Myers-Briggs Type Indicator (MBTI) is a self-report personality survey 
created by Katharine Briggs and Isabel Briggs Myers in the 1 920s. The MBTI is based 
upon Carl Jung’s model of personality and is the primary method for measuring Jungian 
personality types. The MBTI measures introversion and extroversion and is used for 
research purposes as well as career counseling. The MBTI requires several hours to 
administer and evaluate and requires a trained psychological professional to interpret the 
scores. 

The Minnesota Multiphasic Personality Inventory (MMPI) is another frequently 
utilized personality assessment tool. The MMPI determines personality traits of 
hypochondriasis, depression, hysteria, psychopathic deviate, masculinity-feminity, 
paranoia, psychasthenia, schizophrenia, hypomania, and social introversion. The MMPI 
is primarily used by clinical psychologists as a diagnostic tool for assessing personality 
disorders, but is also utilized as a vocational tool. Unfortunately, like the MBTI, the 
MMPI is extremely long to administer and requires special training to interpret the 
results. 
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Projective testing methods are primarily utilized for assessing disturbed 
individuals. When presented with an ambiguous stimulus, such as a Rorschach inkblot, 
the patient will project personal needs, values, and fears onto the stimulus description. 
Projective techniques suffer from low reliability and validity due to the subjective nature 
of the result evaluations [SCHU90]. 

Behavioral assessment procedures evaluate a person’s behavior to a specific 
situation. Researchers assessing the personality of an entire group of people primarily 
utilize this method. For example, hospital staff will routinely observe patient behavior in 
order to identify behavioral trends in patients. This method requires specifically trained 
observers and is highly subject to observer bias, resulting in lower reliability and validity 
[SCHU90]. 
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III. APPARATUS 



A. NEO FFI 

The modeling of an expert CO response to an UNREP evolution is theoretically 
possible if the individual differences of each CO can be ascertained. Collecting accurate 
data about individual differences of COs requires selecting the correct personality 
assessment tool. Most COs are limited on time and relatively unsupportive of academic 
endeavors that take away from their operational duties. While the assessment tool must 
be highly reliable and valid, because of the population being examined and the purposes 
of this research, it must also be easy to administer, easy to complete with minimal time 
requirements, and easy to evaluate with little training required. 

While the MMPI is a predominantly used objective test for assessing personality, 
it is primarily used for assessing personality disorders [BERN94]. The MBTI is a widely 
utilized personality inventory implemented in career related management, but requires a 
trained psychological professional to administer and is too time intensive for the purposes 
of this research [SCHU90]. Though there are a number of objective tests designed to 
measure a broad range of personality variables in a normal population, an increasingly 
popular choice is the Neo Personality Inventory (NEO PI-R) [BERN94]. The NEO PI-R 
is a prime choice for inventorying an expert’s personality since it is the predominant 
measure of the five factor model of personality [WIDI97]. The NEO PI-R consists of 240 
statements to which a person indicates an extent of agreement on a 5-point scale. The 
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NEO PI-R is often referred to as a lexical five factor model since it attempts to define 
personality in natural language terms. 

The “majority of academic psychologists, increasingly favor the NEO PI-R for 
assessment and research” [FURN96]. Furthermore, substantial research exists regarding 
NEO PI-R reliability and validity. Most important, the NEO PI-R has demonstrated 
consistent convergent and discriminant validity, as well as indicating how alternate 
models can be understood from the perspective of the five factor model [MCCR89]. 
Multiple studies have correlated established measures with the NEO PI-R to establish 
overlap, including the Eysenck Personality Inventory and Myers-Briggs Type Indicator 
[FURN96]. 

The creators of the NEO PI-R re-evaluated the usefulness and applicability of 
their personality assessment test. Their findings indicated “far more evidence of its 
comprehensiveness, universality, and practical relevance today than when the NEO-PI 
was first published” [MCCR97]. The NEO Five Factor Inventory (NEO-FFI) is a brief 
60-question subset of the full 240 question NEO PI-R. The NEO PI-R’s additional length 
allows for more precise measurement and better false answer detection while the NEO- 
FFI shorter length accommodates a quicker administration time for the participant. Since 
the creators of the NEO PI-R do not envision any significant changes in the structure of 
the NEO PI-R in the near future, it is a logical conclusion that there are no major 
revisions planned for the NEO-FFI. 

The NEO-FFI scales show correlations with the NEO PI-R ranging from .75 to .89 
for each of the five factors. As subsets of the NEO PI-R domain scales, the NEO-FFI 
scales carry portions of the demonstrated validity of the full scales. While the NEO-FFI 
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scales are not equivalent to the full scales of the NEO PI-R, the shorter scales are 
approximately 85 percent as accurate as the full scales. In the case of the abbreviated 
scales of the NEO FFI, some precision is traded for speed and convenience [MCCR92], 

Previous research inventoried five senior US Navy Surface Warfare Officers for 
individual differences to determine variability in personality using the NEO-FFI [NPS99]. 
Participants consisted of five Unites States Navy Commanders, military pay grade of O- 
5, all of which have been designated Surface Warfare their entire careers. Four of the five 
had served in Executive Officer positions as their last sea going billet, and one had served 
as a Commanding Officer. All five participants scored in the low category for 
Neuroticism with little variance. On average, the participants were high in Agreeableness 
and Extraversion and average in all other categories. The participants exhibited large 
variances in Openness and Agreeableness scores. 

This previous research justified the choice of the NEO-FFI as the tool to assess 
the personality of expert ship drivers [NPS99]. All participants clearly understood the 
standardized directions and had no questions. During the pilot experiment, the inventory 
was easy to administer and on average took less than 10 minutes to complete. Evaluation 
of the results required minimal time and were very easy to interpret by the researcher who 
had no formal personality assessment training. 



B. EXPERT SHIP HANDLING SURVEY 

As part of developing initial profiles for a VCO for COVE, a ship handling 
background questionnaire was utilized. While the questionnaire was primarily for 
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demographic purposes, it did attempt to elicit participant opinions about how to train a 
junior ship driver. The questionnaire was combined with ship driver interviews in order 
to determine the basic VCO framework [TENN99]. 

Previous research surveying and analyzing ship-driving Commanding Officers 
highlighted traits and characteristics that are the same as the traits exhibited by experts 
[TENN99]. Based upon this commonality, experienced ship-drivers were identified as 
experts. Expounding upon the original research, this investigation also utilizes a survey. 
In the early phases of research development, it was determined that a large population of 
expert participants was desired to ensure that the full spectrum of ship driving styles was 
approached. Unfortunately, because of logistical constraints, any form of physical 
interview was impractical. Therefore, the investigator decided to utilize a survey for the 
primary method of ship driving style elicitation. 

Survey questions were primarily based upon previous research examining 
individual differences and ship driving style [NPS99]. In the previous research, the 
participating expert ship handlers were administered an open ended interview regarding 
UNREP. The expert participants were encouraged to state what the key factors were 
when they evaluated novice ship handlers. The results of these expert interviews build 
the fundamental core of this research’s survey. 

However, utilizing a questionnaire for the ship driving style elicitation posed 
challenges that required significant consideration. Since the survey was to be performed 
remotely by the ship handling evaluator participants, the survey must be extremely clear 
since the researcher would not be present to make any clarifications. The size of the 
population of expert ship-handlers precluded qualification of a participant as an expert for 
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reasons other than experience and position. Also, the questionnaire had to be concise 
since most expert ship handlers have limited time to diverge from their duties of running 
a warship. The goal was to maintain the expert ship handling survey completion time to 
under 15 minutes. 

The survey format primarily utilizes multiple choice or rating questions to elicit 
the desired information from the participant. Rating questions were specifically chosen 
because they produce an actual or absolute value of the trait being measured. This 
required developing the rating scale with equal intervals with an anchor position. These 
traits result in rating questions being easier to write and prone to fewer errors [GA093]. 

Figure 2 details the ratings utilized for the survey. The rating scale was 
specifically developed to minimize respondent error and bias. The list of possible choices 
was set at five since most respondents can only distinguish between five to nine items 
[GA093]. Furthermore, the list was maintained short in order to reduce primacy and 
recency effects, effect where respondents are biased toward the last few items because 
those are freshest in memory of a long list of items. The list of possible rating responses 
was always presented in the same ascending order to facilitate proper understanding of 
each rating and help aid recall. 

Another primary concern with developing a question involves avoiding 
inappropriate questions. The expert ship-handling domain is extremely sensitive to 
perceived right and wrong ways of doing business. For example, this requires avoiding 
questions that might require an answer that is directly contrary to guidance or doctrine. 
Regardless of whether or not the expert disagrees with doctrine, it would be socially 
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unacceptable for the expert to declare that he conducts business in a contrary manner. In 
general survey questions were developed to avoid the following questions that: 

• Are not relevant to the evaluation goals; 

• Are perceived as an effort to obtain biased or one sided results; 

• Cannot or will not be answered accurately; 

• Are not geared to the respondent’s depth and range of information, 
knowledge, and perceptions; 

• Are not perceived by respondents as logical and necessary; 

• Require an unreasonable effort to answer; 

• Are threatening or embarrassing; 

• Are vague or ambiguous; or 

• Are unfair 

[GA093]. 



Not Applicable (N/A) -There is no need to perform this action. 

Applicable (A) - This is a relatively minor action with large room for 
variation of execution. 

Somewhat Important (SI) - An action that must be performed to have 
a successful UNREP approach, but with some room for variation of 
execution. 

Important (I) - This action must be performed well to have a successful 
approach with little variation of execution. 

Critical (C) - It is impossible to successfully complete an UNREP 
approach without performing this action flawlessly. 



Figure 2: Rating scale utilized for the Ship Driving Style Survey 
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In addition to avoiding inappropriate questions, the questions themselves “must be 
direct, orderly, precise, logical, concise, and grammatically correct. They must have 
unity, coherence, and emphasis” [GA093]. 

C. EXPERIMENT PACKAGE 

The experiment package consisted of a single survey consisting of five 
subsections: 

• Introduction and Background. 

• Participant Expert Ship handler Demographic Questionnaire 

• Personality Inventory 

• Expert Ship Handler Survey 

• Conclusion and Comments. 

The overall goal was to maintain the total completion time to less than 30 
minutes, because it was a reasonable amount of time to accommodate completion of the 
survey during a lunch hour or other mealtime underway or in port. Furthermore, 
anonymity and privacy were highly stressed to promote participation and help elicit 
higher quality responses. 

Two forms of the survey were created, an Internet based survey and a traditional 
paper based survey. The layout of the traditional paper based survey utilized Government 
Accounting Office (GAO) survey guidelines for design. For example, font size was 
maintained at 10-point type and text was arranged into two columns to promote ease of 
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reading [GA093], The survey was pnnted front to back to minimize the apparent size of 
the document, reducing the likelihood that a potential expert ship handler would not 
complete the surv'ey due to time requirements. Furthermore, the survey was bound to 
improve appearance and better accommodate the participant. 

The Internet based survey was created in order to allow participants the 
opportunity to complete the experiment without having to return any package via the mail 
system. Content was exactly identical to the paper based version, however some 
formatting was changed to better accommodate viewing on a 1 7 inch computer monitor. 
The visual layout was optimized for an 800x600 pixel screen size. The Internet based 
survey also automated data collection by sorting the participants’ responses into a 
database, removing the possibility of any data corruption by the researcher. This version 
of the survey was created using Microsoft FrontPage 2000® web development software 
with all code generated into Hyper Text Markup Language (HTML). 

Prior to commencement of the experiment, two senior surface warfare officers 
stationed at Naval postgraduate School tested the Internet version of the survey for 
usability and functionality. One usability participant was a senior Navy Commander, 
military pay grade 0-5, who had previously served in the position of Executive Officer of 
a warship at sea. The other participant was a senior Navy Captain, military pay grade 
0-6, who had previously served as a Commodore of a squadron of warships. Changes 
regarding content of the Internet version of the survey were also made in the paper-based 
version of the survey to maintain continuity between the two experiment forms. 

Every expert ship handler requested to participate in the experiment was mailed 
the following items: 
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• Cover letter requesting participation from either Commander 
Surface Naval Forces Atlantic Vice Admiral Giffm (see 
APPENDK A), or Commander Surface Naval Forces Pacific Vice 
Admiral Moore (see APPENDIX B). 

• Addendum to instructions detailing how to complete the Internet 
based version of the experiment survey (see APPENDIX C). 

• A paper based copy of the experiment survey (see APPENDIX D). 

• A pre-addressed return envelope. 



While every United States Navy surface warship possesses Internet capability while in 
port or at sea, every expert ship handler was given the choice of participating via either 
the Internet based or traditional paper based survey version. While possessing the 
capability, it might not be feasible for an expert ship handler to participate in the 
experiment electronically because of operational constraints or material maintenance. 
Furthermore, the dual method of participation accounted for problems with electronic 
participation and also helped prevent requested experts from not participating because 
they were not comfortable with the method of electronic participation. 
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IV. METHODOLOGIES 



A. EXPERT POPULATION CANIDATES 

Along with deciding what to ask, evaluators must decide whom to ask. 

The people questioned must have the information the evaluators seek, they 
must be readily identifiable and accessible, they must be willing and able 
to answer, and they must be representative of the population being 
measured. 

[GA093] 

Since the purpose of this research is to learn about the relationships between 
expert evaluators and novices performing a subjective task, the experiment obviously 
required experts in a position to evaluate novices. Because COVE is the vehicle for the 
research, the target population of this research was the expert ship-handlers in a position 
to train and evaluate junior novice ship-handlers. 

Even though the CO is the ultimate person responsible for all ship operations and 
sets the tone for the conduct of all operations, he is not necessarily the only instructor and 
evaluator. In most circumstances, even during an UNREP, the Executive Officer (XO) 
also plays a vital role in instruction and evaluation of junior novice ship-handlers by 
augmenting the CO as an additional coach or evaluator. While not as common, a 
Department Head (DH) is an occasional additional mentor to the junior ship-handler and 
sometimes provides input to the CO for evaluation of the JO. 
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The experiment primarily targeted a sample of COs and XOs currently serving 
aboard warships in the fleet. DH survey participation was also accepted if the DH was a 
recognized ship-handling expert by his CO or XO. These experts were selected because 
they are currently performing the analyzed task and are most familiar with the current 
doctrine and equipment utilized in the fleet. In addition to being a CO, XO, or DH, the 
expert candidate must be serving on a class of warship that conducts UNREPs as the 
approach vessel. This resulted in the exclusion of small craft such as mine hunters and 
coastal patrol craft. Furthermore, while tankers occasionally perform an UNREP as the 
approach vessel, non-Navy personnel who might not possess the same background as the 
targeted expert population usually operate them. 

Based upon the class of ship criteria, there were a combined total of 171 eligible 
warships between the Atlantic and Pacific naval forces. Since every US Naval warship 
has both a CO and an XO, there were a total of 342 potential ship-handling experts to 
sample from. While a larger sample size of ship-handling experts will be a better 
approximation of the total population of ship-handling experts, a minimum of 30 
experiment participants is required to satisfy the Central Limit Theorem statistical rule of 
thumb [DEV095], 

B. SURVEY ADMINISTRATION 

Experiment packages were assembled and mailed via United States Post Office 
First Class delivery to all 342 ship-handling expert candidates. The candidates were 
allowed approximately three weeks until the beginning of July 2000 to complete either 
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the web based or paper based survey. The administration period was selected to 
accommodate mail time both to and from the ship as well as an adequate time to review 
and complete the survey. 

If the web-based version was completed, the experiment participant was instructed 
to not return the paper-based version. If the participant completed the paper-based 
version, the data was recorded utilizing the web-based survey after verifying that the 
paper-based survey was not a duplicate of an already submitted web-based survey. This 
manual conversion of survey format was performed to accommodate automatic data 
collation and analysis. 



C. ANALYSIS 

All raw survey results were compiled into a single Microsoft Access® database. 
The database file was then exported into a Microsoft Excel® spreadsheet. Once in Excel 
format, the raw personality scores for Neuroticism (N), Extraversion (E), Openness (O), 
Agreeableness (A), and Conscientiousness (C) were computed. 

Questions from the ship-handling evaluation section of the survey were classified 
as one of six distinct types consisting of: 



• INTERACTION questions measuring the expert ship-handler’s 
preference for interaction between the novice JO and other entities. 

Other entities could consist of the other members of the bridge 
team, the replenishment ship, or the expert ship-handler himself 

• ANALYTIC INPUT questions measuring the expert ship-handler’s 
preference for the type of rational based decision information that a 
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junior novice ship-handler should use. Examples of Analytic Input 
are rules of thumb, numerical data from ship’s sensors, and 
standard operating procedures, 

• SENSORY INPUT questions measuring the expert ship-handler’s 
preference for the type of instinctual based decision information 
that a junior novice ship-handler should use. Examples of Sensory 
Input are visual approximations for range, non-numerical or non- 
calculation based rules of thumb, and kinesthetic approximations 
for weather forces. 

• COMMUNICATION questions measuring the expert ship- 
handler’s preference for the type of communications the junior 
novice ship-handler should use. Examples of Communication are 
internal and external communications circuits. 

• UNREP STYLE questions measuring the general expert ship- 
handler’s approach to UNREP and what he expects of the junior 
novice ship-handler. UNREP Style questions also include the 
expert ship-handler’s general interpretation of UNREP guidance 
and doctrine. 



The average response for each of the six groups was calculated for each expert 
ship-handling participant. All data was then converted into an input file for the ARC 
software package, a menu driven statistical analysis tool developed at the University of 
Minnesota for applied regression. 

Once the data package was loaded into ARC, a statistical summary of the data 
package was created. The statistical summary contains information such as mean values, 
minimum and maximum values, median values, and standard deviations. Furthermore, 
the statistical summary contains a matrix of correlation values between the different 
variables of the data package. It is from this matrix that significant correlations were 
retrieved for discussion. 
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V. RESULTS AND DISCUSSION 



A. PARTICIPANT DEMOGRAPHICS 

Of the 342 ship-handling experts polled, 136 experts participated in the survey, of 
which 35 participated via the Internet. Eight surveys were incomplete and not used in the 
data analysis. One survey was completed by a participant who did not meet the criteria of 
a ship-handling expert as defined for this experiment. Eleven surveys were returned too 
late to be included in the data analysis package. The resulting data analysis package 
consisted of a total of 1 1 6 surveys. 

At the time of the survey administration, 65 survey participants were serving in 
the CO billet and 48 were serving in the XO billet, and 2 were serving in the DH billet. 
Of all participants, only two ship-handling experts are female. Table 1 further 
summarizes some demographical information of the 116 analyzed ship-handling expert 
participants. 

On average, the ship-handling expert participants had served under eight different 
COs during their career. Furthermore, the average participant had performed between 50 
and 100 UNREPs during their career with a single ship-handling expert who had 
performed over 300 UNREPs. Eight participants were aviators and have not been surface 
warfare qualified for their entire careers. Major ship classes represented consisted of: 
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• Aircraft carriers, including both nuclear powered (CVN) and non- 
nuclear powered (CV) types, 

• Guided missile cruisers (CG), 

• Destroyers, including both guided missile (DDG) and non-guided 
missile (DD) types, 

• Guided missile frigates (FFG), 

• Transport ships including amphibious assault ships (LHA/LHD), 
dock landing ships (LSD), tank landing ships (LST), and 
amphibious transport dock ships (LPD), 

• Other warships not classified above. 



Figure 3 is a histogram that delineates the frequency of ship types for the 
participating expert ship-handlers. Figure 4 is a histogram that describes the participant 
billet distribution between observed ship types. 





Average 

Observed 


Minimum 

Observed 


Maximum 

Observed 


Standard 

Deviation 


Age (Years) 


40.4 


33 


50 


4.1 


Rank 


Commander 


Lieutenant 

Commander 


Captain 


N/A 


Years Of Service 
As An Officer 


18.1 


11 


28 


3.9 


Years Of Service 
At Sea 


10.5 


5 


22 


3.1 



Table 1. Demographic Information for Ship-Handling Expert Participants. 
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Figure 3. Frequency of Participant Ship-Handling Expert Ship Types 
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Figure 4. Frequency of Billet Distribution Among 
Participant Ship-Handling Expert Ship Types 
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B. 



PERSONALITY INVENTORY RESULTS 



Table 2 provides a statistical summary of the observed expert ship-handler XEO- 
FFI results. Figure 5 highlights the personality differences between the average 
participant expert ship-handler and the average over 18 year-old adult participants as 
defined within the XEO-FFI manual [MCCR92]. Means and standard deviations are 
contrasted for each personality trait. Xotable differences are evident in Xeuroticism, 
Extraversion, and Conscientiousness. 
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33.00 


27.00 


34.00 1 


36.00 


Median 1 


10.50 


35.00 


27.00 1 


33.00 i 


39.00 



Table 2. Statistical Summary' For Participant Expert Ship-Handler NEO-FFI Results 



Table 3 delineates the personality differences amongst the participant expert ship- 
handlers. Personality scores were relatively consistent with the exception of carrier 
expert ship-handlers. On average, carrier ship-handling experts are., significantly lower in 
Xeuroticism and higher in Openness and .Agreeableness. These personality differences 
could be related to the demographic difference between carrier ship-handlers and other 
surface ship-handlers since all carrier COs and XOs are aviators. The only other 
demographic group to show slight deviations from the whole was the cruiser ship- 



40 



handling experts who topically scored slightly lower in Xeuroticism and higher in 
Extraversion than the average participant expert ship-handler. 



50 - 




D Expert Ship-Handlers D Typical Adult 



Figure 5. A Comparison of Observed Expert Ship-Handler Means 
and Standard Deviations to Typical Adults As Defined By The 
NEO-FFI Professional Manual. 
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Table 3. Mean Participant Expert Ship-Handler NEO-FFI 
Results For Each Major Participant Ship Class 
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In order to compare a participant’s raw personality score to an average distribution 
of adults, the raw score is converted into a standardized classification group. The 
standardized personality classification groups consist of ver\' low, low, average, high, and 
very high. Ranges for each of the classifications depend upon which score is being 
classified. Table 4 contains the standardized values for each personality classification 
range as defined in the KEO-FFI manual [MCCR92]. 
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Table 4. NEO-FFI Standardized Raw Score Classification Ranges 



Figure 6 is a histogram depicting the frequency of Neuroticism scores amongst the 
participant expert ship-handlers. The distribution is positively skewed indicating that a 
majority of the participant expert ship-handlers possess low Neuroticism. Figure 7 is a 
histogram that illustrates the distribution of Neuroticism classifications amongst the 
participant expert ship-handlers. Figure 7 confirms that 97% of the participants possess 
average or lower than average Neuroticism as defined in Table 4 [MCCR92]. 
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Figure 6. Frequency of Neuroticism Raw Scores 
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Figure 7. Distribution of Standardized Neuroticism Scores 
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Figure 8 is a histogram highlighting the distribution of participant expert ship- 
handler E.xtraversion scores. Figure S illustrates a negatively skewed distribution 
indicating that most participants possess large scores for Extraversion. Figure 9, a 
histogram delineating the breakdowm of the Extraversion raw scores, communicates that 
most participant expert ship-handlers were higher than average in Extraversion. 
Approximately 80% of the participant expert ship-handlers possess a high or very- high 
Extraversion personality characteristic. 

Figure 10 is a histogram that displays the frequency of the observ'ed Openness raw 
scores. Unlike Extraversion and Neuroticism, the participant expert ship-handlers appear 
to possess a symmetric distribution of Openness personality characteristics. Figure 11, 
the distribution of Openness classifications amongst the participants, appears to center 
around the average with an approximately normal distribution. 36% of all participant 
expert ship-handlers possess an average Openness personality characteristic while only 
28% possess a low characteristic and only 22% possess a high Openness characteristic. 

Figure 12 is a histogram illustrating the frequency of Agreeableness raw scores. 
Similar to Openness, Agreeableness also appears to be symmetrically distributed amongst 
the participant expert ship-handlers. However, as Figure 13 details, the majority of 
participants possess an average level of Agreeableness while only 27% possess a lower 
than average level and only 32% possess a higher than average Agreeableness personality 
trait. 
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Figure 8. Frequency of Extraversion Raiv Scores 



53 




Figure 9. Distribution of Standardized Extraversion Scores 
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Figure 10. Frequency of Openness Raw Scores 
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Figure 1 1 . Distribution of Standardized Openness Scores 
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Figure 12. Frequency of Agreeableness Raw Scores 



47 




Figure 13. Distribution of Standardized Agreeableness Scores 
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Figure 14 is a histogram that displays the frequency of Conscientiousness raw 
scores. Conscientiousness appears to have a slightly negatively skewed distribution 
amongst the participant e.xpert ship-handlers. Figure 15 showing the distribution of raw 
score classifications for Conscientiousness shows that 88% of all participants possess an 
average to ver\' high Conscientious personality trait. 
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Figure 14. Frequency of Conscientiousness Raw Scores 



48 



39 



39 




Figure 15. Distribution of Standardized Conscientiousness Scores 
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c. 



SHIP HANDLING EVALUATION SURVEY RESULTS 



Figure 16 is a histogram that summarizes the overall participant expen ship- 
handler's preference for interaction. In general, the experts viewed the novice JO's 
ability to interact with other entities as a relatively import criteria for their evaluation of 
the JO's performance. 65% of all respondents felt that how the JO interacts is at least an 
important criterion for evaluation. Furthermore, these experts desired to coach their 
novices through the evolution via continuous input and feedback and their primary 
measure of interaction is how well the novice JO maintained close verba! communication 
w’ith the expert ship-handler coach. 
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Figure 16. Distribution of Interaction Responses 
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Figure 1 7 is a histogram that summarizes the overall participant expert ship- 
handlers' preference for communication. The majority of expens did not feel that the 
novice JO's personal ability to communicate with the Replenishment ship was imponant 
to their evaluation of the JO’s performance. Most expert ship-handlers feel that someone 
other than the novice JO performing the UNREP should handle personal communications 
between the approach ship and replenishment ship. Only 1 7% of all participants 
expressed communication as an important criteria for LlsREP perfonnance evaluation. 
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Figure 17. Distribution of Communication Responses 

Figure 18 summarizes the participant expert ship-handlers’ preference for 
analytical input. In general, most participant expert ship-handlers believe that a must be 
able to efficiently receive and process analytical information. While all participants 
believed that the novice JO’s ability is at least somewhat important, 67% of all 



51 



respondents fell that anaKiical input is at least an imponant, if not critical component of 
evaluating the novice's performance. 
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Figure 18. Distribution of Analytical Input Responses 

Figure 19 summarizes the participant expert ship-handlers’ preference for the 
novice JO to understand and efficiently process sensory input. All participants believed 
that the novice JO’s ability to demonstrate an understanding of sensory input was at least 
somewhat important to the expert’s overall evaluation. 35% of all participant expert 
ship-handlers view reaction to sensory information as a critical component of a successful 
UNREP and use the JO’s response to sensory information as a major element of UNREP 
performance evaluation. 
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Figure 19. Distribution of Sensor\' Input Responses 

Figure 20 is a histogram that provides a breakdown of how each individual 
participant expert ship-handler approaches UNREP. Figure 20 displays a uni-modal 
symmetric distribution with the majority of participant expert ship-handlers taking an 
attitude towards UlsTlEP that is neither too flexible nor too strict. In general, most 
evaluators allow some deviations from their execution preferences by the novice JO. 
Only 7% require the novice to perform the evolution exactly as the expert desires while 
only 5% of all participant experts allow the JO to perform the UlsTlEP in any safe 
manner. 

The participant expert ship-handlers who tended tow’ards a looser LTsREP style 
placed less emphasis on time to perform the approach as a criterion for performance 
evaluation. In contrast, those experts who possess a more rigid UNHREP style place more 
emphasis on time as an evaluation criterion. Regardless of LNIREP style, most 
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participant expert ship-handlers believe that as expenence increases, time to complete the 
evolution will decrease. 

Approximately 50% of all participant expert ship-handlers believe that UXREP 
documentation and doctrine provides instruction that must be strictly adhered to. The 
other 50% of participant expert ship-handlers interpret UXREP documentation and 
doctrine as guidance that provides a flexible fram.ework for execution. 
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Figure 20. Distribution of UNREP Stvles 

D. OBSERVED CORRELATIONS 

Table 5 contains the intercorrelations observed between the five personality 
factors measured by the N'EO-FFI. Table 6 contains the average intercorrelations for the 
NEO-FFI [MCCR92]. In general, the personality traits were more intercoirelated for the 
participant expert ship-handlers than for the average NEO-FFI participant. The only 
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observed exceptions where the participants possessed lower than average intercorrelations 
were Neuroticism / Conscientiousness, Extraversion / Openness, and Agreeableness / 
Conscientiousness. 
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Table 5. Observed NEO-FFI Intercorrelations for 
Participant Expert Ship-Handlers. 
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-0.21 
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Extraversion 






0.40 


0.04 


0.27 


Openness 








0.02 


-0.02 


Agreeableness 










0.24 



Table 6. NEO-FFI Intercorrelations for 
Average Adults from the NEO-FFI Manual [MCCR92]. 
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In general statistics, a large correlation exists if the correlation coefficient (p) is 
greater than or equal to 0.8 and a small correlation exists if p is less than or equal to 0.5. 
However, the intricacies of human behavior make analyzing real people more art than 
science. Single sample t-Test with a probability of type I error (a) equal to 0.10 indicate 
that correlations with p greater than 0.10 are significant for the participant expert ship- 
handler sample [DEV095]. Therefore, for the purposes of this research, any correlation 
greater than or equal to 0. 10 was identified as a potentially significant correlation. 

Figure 21 graphically displays the calculated correlations between observed expert 
ship-handler Neuroticism values and measured UNREP evaluation criteria. In general, 
correlations were small in magnitude. All observed correlations were negative with the 
exception of the correlation between Neuroticism and Communication, which was only 
slightly positive. 

Figure 22 contains information describing the calculated correlations between 
expert ship-handler Extraversion values and measured UNREP evaluation criteria. In 
contrast with the Neuroticism correlations, most Extraversion correlations were positive 
with the exception of the correlation between Extraversion and UNREP style. The only 
apparent significant correlation was observed between Extraversion and Sensory Input (p 
= .18). This relationship could possible be explained by Sensory Input satisfying the 
extroverts need for stimulation fi’om his surroundings [MCCR92]. 
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Figure 21. Correlations between Neuroticism and UNREP 
Evaluation Criteria for Participant Expert Ship-Handlers 




Figure 22. Correlations between Extraversion and UNREP 
Evaluation Criteria for Participant Expert Ship-Handlers 
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Figure 23 highlights the correlations calculated between the observed expert ship- 
handler Openness personality characteristic and the measured UNREP evaluation criteria. 

All calculated correlations for Openness were positive. Furthermore, all correlations 
were only slightly positive with the exception of the correlation between Openness and 
Analytical Input (p = .16) and Openness and UNREP Style (p = .16). This result might 
be related to the modest association between Openness and measured intelligence since 
Analytical Input is more calculation based. Therefore, it is possible that an expert that is 
higher in Openness would tend to more analytical or cognitive intensive methods. 
However, it must be clear that Openness is not a measure of analytical sense or actual 
intelligence [MCCR92]. 

Figure 24 delineates the correlations calculated between the observed expert ship- 
handler Agreeableness personality characteristic and the measured UNREP evaluation 
criteria. All calculated correlations were positive with the exception of the correlation 
between Agreeableness and Communication, which was slightly negative. The only 
apparent significant correlation was observed between Agreeableness and UNREP Style 
(p = .16). A potential explanation for this correlation is that the highly agreeable person 
is sympathetic to others and eager to help, resulting in an UNREP style that is more 
malleable to the trainee novice [MCCR92]. The expert who scores high in Agreeableness 
sees the UNREP as a training evolution for the novice and therefore has a less rigid 
UNREP style resulting in greater variability and less structure in order to let the novice 
leam fi'om exploration. 
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Figure 23. Correlations between Openness and UNREP 
Evaluation Criteria for Participant Expert Ship-Handlers 




Figure 24. Correlations between Agreeableness and UNREP 
Evaluation Criteria for Participant Expert Ship-Handlers 
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Figure 25 graphically displays the correlations calculated between the observed 
participant expert ship-handler Conscientiousness personality trait and the measured 
UNREP evaluation criteria. Similar to the Opeimess correlations, all calculated 
Conscientiousness correlations were positive. However, two significant correlations 
existed between Conscientiousness and Analytical Input (p = . 1 7) and Conscientiousness 
and Sensory Input (p = .39). The correlation between Conscientiousness and Sensory 
Input was the highest observed over the entire experiment. Experts who score high in 
Conscientiousness are usually purposeful and strong willed and are often associated with 
academic and occupational achievement [MCCR92]. These facts can be directly linked 
to the expert ship-handler’s priority to properly processing all forms of information input. 
The highly conscientious expert will not be able to ignore any information that may 
affect the UNREP, regardless of source. 




Figure 25. Correlations between Conscientiousness and UNREP 
Evaluation Criteria for Participant Expert Ship-Handlers 
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Correlations were also calculated to determine if there was any relationship 
between the participant expert ship-handlers’ NEO-FFI personality traits and Tenney’s 
VCO profiles [TENN99]. Figure 26 illustrates the relationships between the measured 
participant personality traits and Tenney’s Passive VCO profile. Significant correlations 
existed between the Passive VCO profile and the Agreeableness (p = -.25) and 
Conscientiousness (p = -.11) personality traits. In direct contrast, Tenney’s Proactive 
VCO profile correlated with Neuroticism (p = -.20), Extraversion (p = .13), and Openness 
(p = -. 1 1). Figure 27 illustrates these correlations. 




Figure 26. Calculated Correlation Between Observed Participant Expert 
Ship-Handler NEO-FFI Personality Traits and 
Tenney’s Virtual Commanding Officer Passive Profile. 
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Figure 27. Calculated Correlation Between Observed Participant 
Expert Ship-Handler NEO-FFI Personality Traits and 
Tenney’s Virtual Commanding Officer Proactive Profile. 



E. DISCUSSION 

Due to the assistance of Vice Admiral Giffin and Vice Admiral Moore, 40 percent 
of all expert ship-handlers queried responded to the survey. This relatively high 
participation rate greatly increased the quality of the data package by producing relatively 
normal distributions for all areas analyzed. Indications are that the sample of participant 
expert ship-handlers is a decent approximation of the expert ship-handling population. 
Furthermore, the wide range of responses created a broader depth of elicitated ship- 
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handling knowledge that will directly translate into a better database for COVE 



programmers. 

Analysis of the participant expert ship-handler data package indicates that the 
surveyed experts possess some significant differences from the average adult. In general, 
experts were lower in Neuroticism, and higher in Extraversion and Conscientiousness. It 
is clear that whenever modeling an expert ship-handler, close attention needs to be 
focused towards the expert’s level of Neuroticism, Extraversion, and Conscientiousness 
to ensure that the expert behaves in a manner that is consistent with expertise. Future 
research will verify that these personality characteristics apply to all experts regardless of 
the domain of expertise. 

It is important to note that there is still room for variability in an expert model’s 
personality. While the means for participant expert ship-handler levels of Neuroticism, 
Extraversion, and Conscientiousness were significantly different fi"om the average adult 
population. Figure 5 illustrates that the standard deviations for the participant expert 
ship-handlers’ personality traits were very close to the average adult standard deviations. 
This indicates that experts can vary in personality just as much as the average person 
does, just within a different range. 

In addition to the model of expert personality, close attention should be focused 
towards the dynamic of expert evaluation of a novice. All ship-handling experts are 
going to evaluate novice performance based upon a unique combination of evaluation 
criteria. The evaluation criteria create a performance checklist that the novice is 
evaluated against and will consist of a combination of Interaction Responses, 
Communication Responses, Analytical Input Responses, Sensory Input Responses, and 
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the evaluator’s UNREP Style. The emphasis and priority of this novice report card is 
related to the evaluator’s individual personality. Analytical Input will be stressed if the 
expert evaluator is relatively high in either Openness or Conscientiousness. Sensory 
Input will be a critical criterion if the expert possesses a relatively large amount of 
Extraversion or Conscientiousness. Finally UNREP style will be a major performance 
criterion if the expert ship-handler performing the evaluation scores high in Openness or 
Agreeableness. 

Analysis of the data package also indicates that there are links between Tenney’s 
VCO profiles and the model of expert personality proposed in this research. Experts who 
are relatively low in Agreeableness and Conscientiousness will tend towards a more 
passive style of ship-handling as defined by Tenney’s previous work. Furthermore, and 
in direct contrast, expert ship-handlers who display relatively high Extraversion and 
relatively low Neuroticism and Openness will tend towards Tenney’s proactive VCO 
profile. 

With the proper composition of personality traits, and proper combination of 
evaluation criteria and overall ship driving style, a relatively robust model of an 
experienced evaluator of ship-handling is developed. The model is robust because it 
covers a relatively wide range of personality inputs resulting in a potentially wide range 
of unique prototype expert ship-handlers. With proper development, the model bom fi’om 
this research could potentially grow into an accurate representation of human mentoring. 
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VI. CONCLUSIONS 



A. SUMMARY OF WORK 

Understanding how experts evaluate performance of tasks with large degrees of 
variability is a difficult xmdertaking. Just like the tasks themselves, the evaluation does 
not always have the same clear-cut criteria for each expert performing the evaluation. 
Often, critical evaluation criteria are as individualistic as the evaluating individual. 

The primary purpose of this thesis was to assess the relationship between an 
expert evaluator’s personality and the evaluation criteria the expert employed. COVE’s 
UNREP trainer, currently under development by NAWC-TSD, provided the ideal vehicle 
for this research since UNREP is a highly dynamic and challenging task performed by a 
novice whose performance evaluation is oftai a combination of unique objective and 
subjective criteria dependent upon the evaluating expert. 

In order to obtain information about the relationship between evaluation and 
personality, a review was performed on various topics including facets of personality, 
decision-making, individual differences, differences between experts and novices, and 
personality measurement. This review was a crucial step for selecting the proper 
personality assessment tool. Next, an experiment package was developed utilizing a 
questionnaire format in order to elicit information about personality, ship-handling style, 
and novice performance evaluation by a ship-handhng expert. 
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After the experiment package was developed, the survey was administered to 
ship-handling experts resident in the fleet. Support fi"om both Commander Surface Naval 
Forces Atlantic, Vice Admiral Giffin, and Commander Surface Naval Forces Pacific, 
Vice Admiral Moore was critical in ensuring adequate participation by busy ship- 
handling experts. With their help, 116 experts participated in the survey either via a 
paper based or web based version. 

After the end of the data collection period, the participant surveys were collated 
into a data analysis package. The data was analyzed for statistical summary information 
such as means, minimum values, maximum values, mode, median, and standard 
deviations. The data was also analyzed for significant correlations existing between 
participant expert ship-handler personality traits and participant expert ship-handler 
evaluation criteria and ship driving style. Correlations were also analyzed between 
personality and Tenney’s VCO profiles. 

Understanding performance evaluation is a goliath work in progress. Results of 
this research, while beginning to explore the gray areas of variability in evaluation, will 
provide direct enhancement to the training potential of COVE by increasing the fidelity of 
COVE’S rrS. The insights gained firom the expert ship-handler personality traits will 
increase the social realism of the ITS and therefore improve the manner of feedback given 
to the trainee since the ITS can now respond in a more human manner. Furthermore, the 
compilation of critical evaluation criterion and its relationship to personality will enhance 
the task training ability of COVE since it will allow the ITS to train in a manner similar to 
fleet experts. Finally, the creation of a ship-handling database for various fleet expert 
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ship-handlers will provide a solid footing for the overall future development and 
performance of COVE, increasing its quality as a training tool and its utility to the fleet. 



B. THESIS QUESTIONS 

The following questions were addressed in this thesis: 

• Is there a relationship between one’s personality and one’s 
expertise? 

Analysis of the NEO-FFI personality inventory administered to the 
participant expert ship-handlers indicates that there is a relationship 
between personality and expertise. In general, inventoried experts 
were less neurotic and more extroverted and conscientious than the 
average adult. Inventoried experts possess levels of Openness and 
Agreeableness that do not deviate from expected norms for the average 
adult. 

• If such a relationship exists, can it be quantified? 

The model of an expert’s personality should possess the same levels of 
Openness and Agreeableness observed in the average adult, centering 
primarily around average with an approximately normal distribution. 
When modeling an expert’s personality, Neuroticism should range 
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from between very low to average with only a small percentage of 
experts possessing higher than average Neuroticism. In contrast, the 
expert personality model should possess levels of Extraversion 
primarily ranging between high and very high with a lower percentage 
of average extroverts. Finally, the expert personality model should 
possess a primarily average or high level of conscientiousness. 

• Does it extend beyond individual expertise to the expert’s 
evaluation of others’ performance? 

Analysis of the ship handling evaluation portion of the expert survey 
indicates that there is a relationship between personality and the 
evaluation of performance. Evaluation of the novice is based upon a 
combination of factors including proficiency of Interaction, 
Communication, processing of Analytical and Sensory Input, and level 
of adherence to the evaluator’s stylistic tendencies as displayed by the 
novice. The combination of these factors is unique for different 
experts and is dependent upon the expert’s personality profile. 

• What is the range of characteristics of different ship driving styles? 

The data collected during the ship-handling survey spans a wide range 
of ship-handling variability and will be collated and sent to NAWC- 
TSD COVE programmers for analysis. 
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The overall data gathered for this experiment serves as a basis for the 
programming of an ITS for COVE. With this research, a database now exists where a 
programmer can create a model of ship-handling expertise that is highly variable 
depending upon user input. If done correctly, the user could configure a personality 
profile that results in a VCO that interacts, evaluations, and provides feedback to the 
trainee in a way that is completely unique to the personality configuration. 

The relationship between personality, expertise, and evaluation proposed in this 
thesis will allow the ITS to perform in a more human-like manner, increasing the illusion 
of immersion for the trainee, and provide the trainee with accurate performance 
evaluation criteria, increasing the positive training transfer benefits. In addition to 
information about expertise and evaluation, specific information was gathered regarding 
preferences for acceptable UNREP performance. This information will aid programmers 
in determining acceptable limits and bounds for trainees to perform within. 

An important point is that this research is entirely limited to COVE and UNREP 
evolutions. While this research utilized surface ship-handling and COVE as a vehicle, 
the real road is how experts subjectively evaluate highly aggregate tasks such as UNREP. 
The theories and model of expertise provided by this research can potentially span across 
any domain of expertise, as long as the expertise is not composed of black and white 
levels of performance and evaluation. 
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c. 



RECOMMENDATIONS FOR FUTURE WORK 



Tenney’s initial research, combined with the efforts of this research, is a start for 
improving the man-machine interface of tomorrow’s computer based trainers. 
Continuance of this research could vector into numerous directions. Some possibilities of 
future work include: 



• Further investigation between Tenney’s VCO profiles and their 
relationship to an expert’s personality. 

• A further analysis of the data obtained fi'om this experiment 
utilizing expert ship class as a variable 

• Examining the dynamic between expert and novice accounting for 
the personality of the expert as well as the personality of the 
novice. 

• A linear, or non-linear, regression model of evaluation with 
coefficients based upon the correlations proposed in this research. 

• An agent based autonomous agent, such as STEVE, that possesses 
a configurable personality within the bounds of expert personality 
as defined by this research. 



Understanding the intricacies of human behavior and its relationship to highly aggregate 
tasks that are subjectively evaluated is a mammoth effort. Hopefully, this is only the 
begiiming of a series that attempts to quantify how humans act and react. Performance of 
this type of research is a critical step in increasing the utility of computers by making a 
computer more like a human instead of a human more like a computer. 
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APPENDIX A. COMMANDER, SURFACE NAVAL FORCES ATLANTIC 



ENDORSEMENT 




i-rviN L/H 



:r naval surface force 

ATiANTlC FLEET 



8 May 2000 



I 



Dear Commanding Officers, 



The Naval Postgraduate School (NPS) maintains a proud 
tradition of serving as a "'Tech Bridget accommodating advanced 
research in direct support of Na\y goals and future development. 
The Conning Officer Vinual Environment (COVE) program is a 
prime example of how NPS research strives to take Technical back 
to Tactical. Cove is a surface ship-handling trainer that utilizes 
cutting edge computer graphics and physically based modeling to 
simulate the ship driving experience. COVE is scheduled for 
imminent release to S WOS and the fleet. C A 



The enclosed sur\*ey is part of an NPS thesis that directly 
supports COVE. The surv'ey seeks to increase the fidelity and 
realism of interactions between the COVE trainer and the student. 



By completing this sur\'ey. your time will improve the quality of 
this 21''"^ century readiness tool. 

Sincerely. 

\>yCvL/v ^ 




HENRY C. GIFFIN III 
Vice Admiral. U.S. Navv 



1 
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APPENDIX B. COMMANDER, SURFACE NAVAL FORCES PACIFIC 



ENDORSEMENT 



i 

j 

i 

1 






COMMANDER NAVAL SLRFACE FORCE 
UJS. PACIFIC FLEET 



I 

I 

I 

I Dear Commanding Officer. 



12 May 2000 



i 

I 

i 

i 



1 

I 

I 



I 



Naval Postgraduate School (NFS) maintains a proud tradition of 
serving as a '’Tech Bridge.” accommodating advanced research in direct 
support of Nav\ goals and future development. The Conning Officer 
Virtual Environment (COVE) program is a prime example of how NPS 
research strives to take technical back to tactical. COVE is a surface ship- 
handling trainer that utilizes cutting edge computer graphics and 
physically based 
is scheduled for 



[ modeling to simulate the ship driving experience^. COVE ^ 

im minent release to SWOS and the fleet. 



The enclosed survey is pan of a NPS thesis that directly suppons 
COVE. The sur\*ey seeks to increase the fidelity and realism of 
interactions between the COVE trainer and the student. By completing 
this surx’cy. your time will improve the quality of this 2P^ century 
readiness preparation tool. 




! 



I 



\ 
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APPENDIX C. EXAMPLE ADDENDUM TO INSTRUCTIONS 



Fro:' • :S 

Vo .'ov:\iA\D.\(;oi-; icFR ; ssbonhommi; Richard 

N.h:. addend; :\str;. ct:ons for r.XPRRT skiphandler ni;rvey 



:•> >v:rvc\ ’^ps; u\;i:iub>c on the ;:^:crnc'.. You c:-.vCi;r.j«cn :o puntcjpat? jr. :hc ueb bc;.*;c<i vcrs.or: 
b> ;hc foiiowing aedresx vmo :hc AccreSN Bar Expiorcr Fr^u'-tr E or ;nto :hc EocatJon Bar 

Coff'^rufu^ aror Ft^.irc 2 j 

‘Javj;. ,ni; ^ oves L \pc~' •?’‘cr.: 

: rsc ACT ^ Nn C'ODIt ii> cn;n into :hc wco site is. Woves 'Vc.vr .voi.v.'/.nc/. 

ehoovc To partic.-pari.' :n the s^co based version. jOi: .ire no: reesuned torclivm the sar\cy packer 
Kejiare'ess of t'omuu picnic vorrn.c:c :Fc sur.'c;. no iarrr !::an July 1. 2C00 



L .:g~ 









-!« P MAS^cr»:5 CYAl-aATC 
N'Ol^Cta 






Figure : 

Exemp.'c I'itlc Screen as v i 
Interne; Explorer 



1 



agg«gf»cr;; 
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APPENDIX D. EXPERT SHIP-HANDLING E^^A.LUATION SURVEY 



■NAVAL POSTGRADUATE SCHOOL 

SURVEY OF HOW EXPERT SHIP HANDLERS 
EVALUATE NOVICES 





r 



1 
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NAVA^ POSTGRADUATE SCHOOL 



:ixi: 






SURVEY OF HOW EXPERT 
SHIP HANDLERS 
EVALUATE NOVICES 



INTRODUCTION 
V/hy Should I Participate? 

Ba>ed upon sour cxf^crtcncc arv. vurrcnt 
you v.cr;r iJciuirjcji un art export shrp 
handler Your participation this 
ensure that ceruin aspects of s.miiiato: training: 
arc rcaiistic and pcnincm and that simubior- 
tniihinc systems are Capable of instrjchaj 
trainees in a similar fishion to how you instraa 
your Junior Offscers 



COVE Simulator 

This survey sopperts development of *»hc 
v'onnine Officer b'lrtuol Environment (C’OVE). 
sh;j>dnvsn^ simulator Ncheduiod for imminent 
release to both S v\*OS and *dic tlceL CCA‘C :$ 2 
fully tanded training project beinp developed by 
NTS. Nu'.al Air 'Warfare Center Training 
Systems Division <\'AWCTSD>. and the Office 
of Na%al Research <ONR>. 

CO\'E allows junior ship drivers the 
o;?jx)ruinjiy to pmaicc dangerous sliip hansiling 
evoiuttons on a shorc-baxid facility \«ih no 
danger to matenrd or personnel. 



Privacy Information 

Your anonymity is maintained a: all times 
throughout this survey, and you: privacy js 
sateguarded under OP\A\ Instniciion 
?i?00,oOB There is no record linking your 
participation with this survey .A record of the 
mformaiion contained in the expenment 
desCTibvxi hcahn w,ilj be retained permar>enily at 
the Naval Postgraduate School or by higher 
authority. Answers provided by your 
participar.on will be used only for .sitatisrical 
aniilysis by ibe Departnxmts of the Navy and 
Defense. at>d c^hcr L’.S. C>ovemnvcnt agencies. 



provided this use is compatible w;th the purpo^^* 
for which the mform.UK'n vs as coilectes.1 

By panicipaii.ng in this su.'vcr.. you 
acknowlcsl^e that you understand iha: your 
privacy is mainiatncd as stated .iN'vc 
lunhermofe. you are acknowledging that your 
participation is completely voluncir.. 



INSTRUCTIONS 
How To Complete This Survey 

VVe know that your u.mc is voluoMc. Ifiis 
survey shc'uld lake apjeovimately 30 minuto to 
complete The survey consists of three ports* 

I > ProfcsMonoI dcmogra;:*ic secoon 

2 1 I^ersonolity inventory 
» L'NRIiP evaluation section 

Completion of this survey will require you 
to either fill in blank response fields or complete 
a multiple choice by circling your ortswer. 
Please answer every question in this survey ic 
the best of your ability If Lberc arc any 
problems with this survey, please contact: 

I.T. Chris Buziok. LSN 

C^xie 32., N'nval Posigniduate SchrK>‘ 

b.'3 Dyer Rood, Rm 4^1)4 

Monterey. CA 93<>43-5000 

(vS3 1 >656-4679 

DSN S7$-46?9 

I pon completion, please return the survey 
in the enclosed envelope 10 the above address. 

Thank you for your time otkI pomciparion. 



’ Email any quesnons to L I . Chns Buziok 2X 
bu/aak<<^ cs.nps.navy.mil 



1 
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PROFESSIONAL DEMOGRAPHICS 



Rank; 



Primary Warfare Specially: 
({Me one* 



Surface 

Aviation 

Submarine 

Other 



Age: 



Sex: 

(t'trrle 



Male 

Female 



l oial Years of Officer Service: 



Number of years tn a sea billet: 

(Total} 



Number of Commanding Officers you 
have served under wtile at sea: 



Current Sea Billet; 
/// at shore. Tdl in SHQREi 



Current Class of Ship: 
(If at m in SHOFIE » 



Time since last ar sea: ? 



j 

I i 


0-50 1 


Approximate Number ^ 


51-100 1 

1 


of CNREP Approaches 


101 -200 ! 


(Total during your career).- 1 




(ctrrlf; onf) i 


200 -?99 i 




More Than 300 1 
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PERSONALITY INVENTORY 
Instructions 



This section of the survey consists of a short. 60 question, personality in%'cntory. This personality inventory is a 
standardized professional psychological tool that \PS is utilizing to Icam more about the experts residing in the 
lleet. NTS believes that accurate modeling of expen leadership will enhance the fidelity’ of the COVE tramcr. 
The inventory takes appwoxunately 10 minutes to complete. Please answer all of the following 60 questions as 
accurately and truth tully as possible selecting an answer that best represents your opinion Try not to read too 
much into any question. Your first impression or thought after reading the question should be your answer. Do 
not go back and change a previous answer Possible answers range from strongly disagree in the leftmost 
column to strongly agree in the nghtraosi column. 

(SO) Strongly Disagrao (w ihe suiemou is definj icty 
(O) D isa gr e e if ihc soienicm ts incKtly tatsc 

(N) Neutral if you cmnot deoeV;. cf ihc $i:iii:iictii u vqiuUy uve lUid false 

(A) Agree rr the satement i* nv-»t!y true 

(SA) Stroftgly Agree or the saicnaa ls Uciimtcly trjo 

Choose only one answer per question. There is no time liout. 



Inventory 



Question 


SD D N 


A 


i SA i 


1 . 1 ;ia) not 3 wvmcr 






i 

j 


2. I like to have 3 lot of people around aac 






! 


5 . I dui\ bko lo waste umcibydn:aTreag 








4. I try to be crurtoous to everyone 1 meet 








5 . 1 keep my tKtongixigs clexn and ocat 






1 

i 


h 1 oflen fed ioferioi to others. 








7, lUiu^ easily. 








8. Once I find ibe right way to do sofnrihing. I stick to it. 








9. I eften get ii«o argumaav with my hiaiity and co-wurVars. 


1 






10. t'm prctiy good 3bout pacing mysell so as to gel dungs done on utne. 








1 1 When rm under a grcai dcaJ of stress, som.iimcs I feci like I’ni going to pieces. 








12. 1 don't coQ.«ider myself especafly "light- beaned". 








1?. I urn imngtaed by the piucms t hnd m an and ouure. 


j 






14. 5U>mc people think I’m sdiish aiuJ egotistical. 


j 






15. J am not a very methodical parson. 








16. I rardy feel alone or blue 


; • 






17. I really enjoy ulking to pcopk. 









3 
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(SO) stronp/y DiM^TMoclb^staiJccDcaiis ddtnjt^lvfaHc 
(0) Drs9ffree if ibe uaurmcnr in mcHily fai;e 

(N) Nautnl if you ctooM dcddc. ur the stAietuexU la equally tnx sod Ulve 

(A) Agr»0<Tt che sutemetu is mouly true 

(SA) Strongfy Agr— a tbc suicncut b (JoTiuliely uue 



QiMStion 


SD 


D 


N 


A 


SA 


IS. 1 bdicvc iCTitng nudcnu iscar controvcrsmi speakers can only confuse aod nalead diem. 












ly. l would railicicooperaiewulJoaicisikmcOTpeie with die^ 












20. I tr>’ 10 pexform ail the task.<; assigned tn roc <xiii.«icuaiiiouRl> 












21 . 1 oftai feel Lem« and jittery. 










22, lUkc to be where cljc action «, 












23. PoetT) has link or no effect ootne. 












24. ItendiobecvnKalaodidccptkaJofodicrs'inteiilionL 












2S. I he>e a clear ^ of und wort toward them iq ao orderly Xathioa 












26 Sooctlixiesl £l^ 2 icomplctLiy »-ti(tlilc». 












27. I lenally prefer to tb thin^ alooe. 












28. loficnto'new and fordptfiTods. 










— 


29. I helicvc thar most people will take arlvantaj^e of yon if ytjti iei them 










30. 1 waste a kt of tune before settUi^ down to work. 












31 . 1 i«fvl> feet rcarful ^jr afictioua. 












32. I often feel as if I*m bummg wah eoetty. 












33. I sddocn notice the norelt re fcchnat that rf flereni enwonments pr(xlv€& 












34. Xlratt people I know like me. 












35 1 work bard u> accomplish my foals. 












3d I often get ai^' at the way people treat me. 












37. 1 am a checrtUl. Mgh-spirlicd person. 












.38. t hclioe we .thnuld 1 <kA tuucr religKius autbodoes for dedsiooK on mxtl isauea. 












39. Some people Hunk of me m cuU ami calcublmg. 












40. >^11(31 1 make a comcilmcm. 1 cn always be enuxtied on to follow- tttrousb. 












41. loo often, when things so wror^ 1 gctdisco«r&t6d end fbd hke giving op. 












42. 1 am o«t a riioerfijl oprimiat. 












43. Sometimes when 1 amreadin^ r^ry nr loobog aia wort: of an. I fed a chill tx wave of 
cxcxtcaami. 
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(SO) Slrongty Divagrw «)r ihc stticiaaii is iJcfuuicly Cilsc 
(0) CUs»gr9^ »f the sutcincm js iDCrsUy lalst 

(N) Neutrat if you cannci dociJe, oc ihe soiemcnt equally inic and false 

(A) Agree cr the state mciu is mastly true 

(SA) Strong)y Agree nr the statcmcot is dcfiojtdy true 



Question 


1 SD 


D N 


A 


SA 


44 Tm hani-hcade4 anJ lough-ncnded m inv olUludcs. | 


I 




* 

45 Someumes I'm nut os dqxcndoblc or reliable as 1 should be { 


1 1 
1 




45 I am sdduio sad or depressed 




i 




47 My life IS fast-paced 




1 1 






48 I have btxlc iniacst m spcculaung un the nature of the umvtasc oc the huiuoa oocHlmuu. 










4^ 1 generally try to he thoughiful and cr«ttidcraic 




, 






50 1 am a productive person who al^Mvs gets the jo6 done 




! 






51 1 oiten feel helpless and vvam srmifonc cLsc to solve my problems. 




1 






52 i am a very active person 




1 






55 1 have a lot 01 intcUoctual cunosity 








i 


M If I (Jim\ like people. I la ihcio know a. 




1 




i 

1 


55. I never seem to be able to get organized 




1 






56. At times 1 have been so ashaiudl just w:aot to bide. 




i 






57. I would rather go my own uny then he a leader of nrhers 




1 






58 I often enjoy playing wsib thconcs or abstxaa ideas. 










If neressory. 1 am wiling to nanipulaie people to get what 1 waia. 










60 I strive for eacdleno; in everything 1 da 




1 
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UNREP EVALUATION SURVEY 
Introduction 

lliis portion of the survey deals with bow you would evaluate a junior ship handler. For the purpose of this 
survey, imagine that you have just supervised a young, less ex pcneoccd junior officer who has completed the 
appro;ich on an AE Following completion of the evoluhon, you are usked with evaluating the JOs 
p^drmance and providing the JO with feedback about the execution of the approach. This portion of the 
survey is intended to determine the specific criteria you use to evaluate a conning ofTiccr s UKREP approach 
performance. 

This inventory consist of 3 pans. While the entire inventory requires only about 15 minutes, there is no time 
limit. Please answer each of the following questions considering your requirements for a successful approach. 



Parti 

Part I requires you to evaluate each of the following 3 UNREP approach templates. Evaluation criteria for 
each of the three UNREP styles are: 

Pref^md • This method rnosi closdy maicbcs hem yoo cxpeci your Conning Officer to itnkc UNREP approach. 
Acc^tablo - This method imy not be exactly bow you expect the approach to occur, but you have no ob)cctioc:> to this type of 
cxecuitou 

Un»CC^jAat>l9 - You will not allow a Conniog Officer to execute the UNREP in this fashion. 



Ltgfnd Far Duzj^ramj 
RECOMMEND (COURSE SPEED! 



j Course 


Speed 


I -Left 


I- lOLTUSSC 


i R-RiKht 


D - Oodcox 


! N - No Chance 


N ' No Chanc 
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1 20 feet 




(cifclf Ofur) 

Preferred Acceptable Unacceptable 
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1 20 feet 




(circle one) 

Preferred Acceptable Unacceptable 



S 
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( 



0 feet 



1000 feet 



1800 
2000 feet 



120 feet 




(cifcie one} 



Preferred 



Acceptable 



UnaccepQble 



g 
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Pari II 



Part II cover? specific criteria that may be used lo evaluate a Conning Officer Possible answers for each of the 
following questions in Part II are; 

Not AppltcMbfe (N/A) ♦'IlKfc u do need u> perfortn ihis action. 

Apf>HcMt>l& (A) - TIils ts a iviauvely minor acuon wUh Ur^ njum Tor varuluxi ol c.\ccmuni. 

So^what Important (SI) - An actuxi ihai musi be perfortneU lo have a succci^lul UNRtF approach, but 
wub some room for vanauoQ of cxccimon. 

Important (}) - This aiimon must be performed wdl lo have a succcssfuJ approach wub Iiule vanaiion of 
tvcculiiai. 

CnticBl (C) * It is tmpossibie lo successfuJIy complete an UN RHP approach without pcrfornnog Um action 
flawlessly 



Please answer each of the following questions by providing one of the above ratings. If your ship does not have 
a required piece of equipment to perform an action, answer as if the required gear was installed on your ship. 



Conning Officer 
Preparations/Qualifications 

WMr you do not n^cemuity expect tfie Conninf; 
Officer to perform these actioru. :fie\ are tjunvledge 
reguiremmis for a ffood Conning Officer 


N/A 


> 

1 S2 

1 

1 


1 ■ : 

1 ; 


1 . Po<»cvsc» an m-dcpUi bvjrwledge of own :dnp‘s 
nuuvnc nues of accelcriUooAlecdcmioo and <xbcr 
handling cfaanictcnstics. 




' 1 * 


j i 

I 

1 _ _ 1 


2 . P<wscsvcs an lu-dcpUi know ledge of handbitg 
charactmstics of atnuliar>’ ship fe.i*. accelemt/on and 
turn nuesj. 








j 


3 . Possesses on undcntandiDg bow Own Ship’s 
vibraoons oerrespood to speed and rudder commands 










4 . Possesses an abilit)* to seir«e enviroamenia] factors 




i 






5 . Possesses an oodecstanding of wind force affects 
on ship's freeboard. 




j 






6 . PinsseKses understanding of depth of ocean effects 




1 






7 . Possesses understanding of topography of ocean 
flcxir eflccts. 










1 


&. Possesses uxKkrsundmg of sea state effects. 










1 


<) Possesses undersumding of direcbon of seas 
effects. 












10 . Possesses understanding of Bernoulli forces 












1 1 . Possesses undoMandiDg of JA/lntegmod ship's 
arcinl. 












1 2 . Possesses undorsunding of radio bcadsct/walbc- 
nUac for uucroal conwamiciuons. 












13 . Possesses understanding of Bridge to Bfidge 
handset 













iO 
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Not Applicable (N/A) -IImv no need to perform thu icooa 

Applicable (A) - This is a rdaiiwdy minor action with large room for variation of cxcoitioa 
Somewhat Important (Sf) - An action that mu«i he performed to have a successful L^NTIFP approach, boi 
with some room for vanauoo of csocv&ua 

Important (I) - This aaion must be performed well to have a succcssfuJ approach with Itnlc variauoo <if 
weeutwa 

Critical (C) - It li irnpossiWc to successfully complete an UNRhP appruacb without perfurmmg this aamc 
flawIcvKly. 



Conning Officer 
Preparations/Qualifications 

While you Jo not neceisartty expec: the Cenrun^ 
Officer :o perform (Krse actions, Jiey are kjioM,leJ$e 
re’jutremenli for a good Conning Officer 


N/A 

i 


A 


! 1 

SI 1 

I 

! : 


C 


1 4 Possocscs understanding of UHF. 






! 1 

i - i 

1 1 

1 : 


1 


1 S Pcvcsesvcs understanding of cellular 
£\>mnminjcauo&. 






l&. !\>s$isscs uihlcntandmg of hgbu r/<>r 
communicatton). 




1 


1 




1 7 Possesses understanding of scnoaphcrcs. 






1 




18 Possesses uiKlcrsunding of formal dcciston aids 
<irtformiuion br>ofiis. cherk^m. flow chr.m e\r > 








1 


1 9. Possesses uodcrstaoding of NWP proa^dure 




1 


i 


1 


20. Possesses understanding of SOPs. 




1 


i 


i 


21 Displayed proper u.sc of MOBOARD. 
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Not Apf>ilcsbt» (N/A) 'There ts do need (o pdform ihn acuoo 

AppUcst>i0 (A) - ThL<i i!( a reiativ’dy minor actum with Ur;ge room for vanation of execution 
Scjrmwtist Importont (SI) - An achoo that must be porfonned to have a successful UNKLP approach, but 
with some room f<T v-ariaiion (A cxccuiioa. 

important Q) - Tlas actioo usisi be pcifanned well u> have a buoccssful approach isilh htUc vanatu.m of 
escctmon. 

Critieaf (C) • It >s impossible to successfully complete ao UNREH appreveb wuhuut pertornuog this icuon 
flaivleisly. 



Execution 

7he.tr or* actions you rjprci a good Canning Officer 
10 perform »W/ during the USREF approach 


N/A 


A 


SI 1 1 


c 


1. MaiDUined dose verbal communicatum between 
Conn and CO/XO durm^ approach. 


- 








i - 1 


2 Maintained rcpilar conununicatioQv between own 
>bp and Auxiliarv dtinii^ approach. 








i ! 


3. Displayed fsvpa use of lace to face 
comituQications. 








i 1 


4 Displayed proper use of J Vlmcgraicd ship’s 
or cun. 








. } 1 


5 Di$pla>-edpropd use of radio hcadset/wallac- 
lalloe fee mlcmiil cuinniumcatiuii.s. 










6 Displayed proper use of Dndge to Bridge handset. 








1 1 


■T DLSpUyed proper of UHf 








' ! 


S Displayed proper use of cellular cuuununicaiioa 








■ 


V Dispbyed proper use of lights (Jor 
commurJeaiion). 








! 

j 


10 Duiplaycd proper use of semaphores. 








\ 

t ' 


11. Displayed proper use of fonzol dcosicm aids 
tinformadon checkksix, fUrw chans etc . ) 










i i 


11 Displayed proper fonzBil use of NWP procedure 










i ! 


13. Displayed proper use of SOPs. 








j 


I 


14. Displayiod proper use of rules of thumb (non- 
numeric, non-calcukuion based) 








i 


f 

! 


15. DLSplayod proper use of radian rale fluid 2 My 
other escalation Sued rule of thumb). 








1 


! 

1 


16. Displaytsd proper use of an inienBl checklist or 
other svstcnamcroucuie for oompletioii of tnilcsiones. 












17. Displayed proper use of a scan cottine for input 












18. Displayed proper visual observatioo of wake to 
' iudge relative motioa 












19. Displayed proper visod observatioo of rdanve 
pmpeciive to ludge rclaiivr cnotioiL 












20. Displayed proper vinol observaboo of rate of 
cKkutc to ludjtc relative motion. 












21 Displayed proper and bmely use of GPS 
po5ttton/sp^ 












22. Displayed proper use of MOBO.ARD. 












23. Displayed proper and bmely use of StadUQctcr 
urfnrmation. 












24. Displayed proper and bmely use of Radar range 
otforanboo. 












25. Displayed proper visud esOxtBUoos of range. 












26 Displayed proper and bmeJy use of laser 
rauteftodor tnfennadon. 










i 
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Part ill 



Part III consists of some general questions about UXREP approaches. Answers for Part III consist of multiple 
choice and fill in the blank that best correspond >vith your expert style of ship driving. 



What IS your mioimum ailo'^'able lateral separation? 

Hnjeet) 




W'bat is your maximum allo^^■able lateral separation? 

fin ftfti 




WTiat is your ideal approach speed differential? 

(in krj9tt) 




WTiat is your ideal distance behind the 
Auxiliary pnor to commencement of approach? 

(tn \QTdft 




What is an adequate amount of ume lo perform an 
approach for a ne^^' Junior Officer? 

(circle one} 


Under 5 minutes 
Between 5 and 10 minutes 

Between io and 15 minutes 

Between 1 5 and 20 mJnmos ! 

Greater Than 20 mb^utes 1 

Time IS a luncOon ol speed and (kstance • 

There is no ‘adequate* amount ol time \ 


Wliat is an adequate amount of tume to perform an 
approach for an experienced Senior Officer? 

(circle one) 


Under 5 minutes 
Between 5 and 10 minutes 

Between 10 and 15 minutes 
Between 1 S and 20 minutes 

Greater Than 20 minutes 
Time is a lufKlJon ot speed and distance 

Ttiere is no ‘adequate* amount ol time 


How would you characterize your managemcot of Conning 
Officers during an UNREP? 

(circie one} 


i allow large vartaoons in 
execution from my preferences 

1 allow some vanations in 
execution from mypr^erences 

1 allow no variations in 
execution from my preferences 
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1 



Which srarcroent best desenbes the NWP? j 
(circie one) j 


NWP is a detailed roc«pe for UNREP 
j NWP is only a tooso framework for UNREP 


1 

Which are you most comfortable with: 
icircle vne) j 


A conning officer who pnmahfy uses rules of 1t>umb. 
1 ship’s instruments and other guidelines to make 
decisions’? 

1 (someone y/ho kncMS numbers and facts) 

or 


1 

i 


A conning officer who pnmartty uses hts instincts and 
sensory esumaaons? 

1 (someone r/ho drives a shtp like a car on the htotrway) 



CONCLUSION 



Thank you for taking your \-aluabk: time to 
complete this surv^ey. goal of this research is to 
deiennioe bow lo tailor feedback provided to a Junior 
Officer (JO) during training in a Virtual Environment 
(VE) ship handling training simulator such as 
NAWCTSD’s Conning Officer Virtual Environment 
(COVE). Tailored feedback allows a simulator to 
train a JO in a way that is compatible \Aiih the JO*s 
Commanding Officer s ship handling style. If a JO 
can receive feedback chat is similar to feedback 
he receives from his shipboard mentor, the JO will 
have more effective simulator training lime and be 
more productive while at sea. 

If you would like to leave suggesuoos or 
comments, free to enter them on the back of this 
survey: 



For additional questions or commcnis about this 
research, please contact: 



LT, Chris Boziak. USN ^ 

Code 32. Naval Postgraduate School 
S33 Dyer Road: Rm 404 
Monterey. CA 93943-5(X)0 
(83 1 ) 656^79 DSN 878-»679 

Please encourage your expert peers to participate 
in this survey. Greater expert participation will 
improve the results and create a better product 
released to the fleet. 



^ Email any questions to LT. Chris Buziak at 
buziak<§'cs.ops.navyjniJ 
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Enter Your Comments Here 
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APPENDIX E. UNREP CHARACTERISTICS BASED UPON 



APPROACH SHIP CLASS 





SMALLEST OBSERVED 
MINIMLFM ALLOWABLE 
LATERAL SEPARATION 
(FEET) 


AVERAGE MINIMUM 
ALLOWABLE LATERAL 
SEPARATION 
(FEET) 


LARGEST OBSERVED 
MINIMUM ALLOWABLE 
LATERAL SEPARATION 
(FEET) 


CG 


80 


117 


140 


CV/CVN 


120 


136 


140 


DD/DDG 


80 


122 


150 


FFG 


60 


109 


160 


Transport Ship 


80 


135 


180 



Table 7. Minimum Allowable Lateral Separation 
Between Approach Ship and Replenishment Ship 
Based Upon Approach Ship Class 





SMALLEST OBSERVED 
MAXIMUM ALLOWABLE 
LATERAL SEPARATION 
(FEET) 


AVERAGE MAXIMUM 
ALLOWABLE LATERAL 
SEPARATION 
(FEET) 


LARGEST OBSERVED 
MAXIMUM ALLOWABLE 
LATERAL SEPARATION 
(FEET) 


CG 


160 


185 


200 


CV/CVN 


180 


200 


220 


DD/DDG 


140 


189 


250 


FFG 


no 


182 


240 


Transport Ship 


140 


203 


300 



Table 8. Maximum Allowable Lateral Separation 
Between Approach Ship and Replenishment Ship 
Based Upon Approach Ship Class 
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SMALLEST OBSERVED 
SPEED DIFFERENTIAL 
(KNOTS) 


AVERAGE OBSERVED 
SPEED DIFFERENTIAL 
(KNOTS) 


LARGEST OBSERVED 
SPEED DIFFERENTIAL 
(KNOTS) 


CG 


5 


9 


18 


CV/CVN 


3 


5 


6 


DD/DDG 


5 


7 


20 


FFG 


5 


11 


25 


Transport Ship 


3 


6 


18 



Table 9. Allowable Approach Speed Differential 
Between Approach Ship and Replenishment Ship 
Based Upon Approach Ship Class 





SMALLEST OBSERVED 
STARTING DISTANCE 
(YARDS) 


AVERAGE OBSERVED 
STARTING DISTANCE 
(YARDS) 


LARGEST OBSERVED 
STARTING DISTANCE 
(YARDS) 


CG 


400 


671 


1000 


CV/CVN 


1000 


1800 


2000 


DD/DDG 


160 


623 


3000 


FFG 


300 


652 


1500 


Transport Ship 


300 


748 


1800 



Table 10. Allowable Starting Distance For Approach 
Between Approach Ship and Replenishment Ship 
Based Upon Approach Ship Class 
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